In this article, we shall primarily focus on the intuition and implementation of the \(\rm{R}{\small EINFORCE}\) algorithm.
First, from last time, we discuseed how to paramaterize the value function \(\hat{v}(s, w)\). Now, we shall directly parameterize \(\pi(s, \theta)\). However, instead of trying to minimize some least-squared-error, we shall instead try to maximize rewards \(\theta \xleftarrow{} \mathrm{argmax}_\theta \ g(\pi, \theta)\).
Then, our update rule becomes
\[ \theta \xleftarrow{} \theta + \alpha g^{(m)}_t \nabla_\theta \ln (\pi(a^{(m)}_t \mid s^{(m)}_t, \theta)) \]note that the \(\ln\) is present in order to normalize the rewards based on the probability of selecting an action.
One issue with maximizing rewards is that the absolute magnitudes of the rewards do not matter. This means that, for example, adding a scalar \(c\) to all rewards would cluster their relative magnitudes. Then, as it is harder to decide between rewards that are all similar, it would slow down convergence to an optimum.
Thus, we shall introduce a bias term, in order to try and center the expected rewards. In particular, one natural choice is to reintroduce our (parameterized) value estimate \(\hat{v}(s, w)\).
\[ \theta \xleftarrow{} \theta + \alpha [g^{(m)}_t - \hat{v}(s, w)] \nabla_\theta \ln (\pi(a^{(m)}_t \mid s^{(m)}_t, \theta)) \]The Policy Gradient Theorem states that the above update rule will converge to a local maxima of the \(v_\pi(s_0) = \mathbb{E}[G_0 \mid s_0]\). In particular, that
\[ \nabla v_\pi(s_0) \propto \sum_s \mu(s) \sum_a v_\pi(s, a) \nabla \pi(a \mid s, \theta) \]again, note that the \(\mu(s)\) term is applied implicitely by the gradient sampling algorithm.
One major advantage of PGMs is that they learn smoothly, where the policy changes are continuous. This is in contrast to estimating a value function and having an \(\mathrm{argmax}\) policy (or \(\epsilon\)-greedy). A major consequence is that it may learn stochastic policies.
Further, as it (generally) skips the need for estimating the value function, it does not learn as much about the environment. This may be beneficial, as it can ignore irrelevant details, or harmful where it discards useful information.